Theory

theory

Tweedie's formula

One can estimate the mean of a Gaussian distribution, given a random variable $z\sim \mathcal{N}({\mu }_z,{\mathrm{\Sigma }}_z)$$z \sim \mathcal{N}(\mu_{z}, \Sigma_{z})$ by $E({\mu }_z|z)=z+{\mathrm{\Sigma }}_z{\mathrm{\nabla }}_z\log p(z)$$E(\mu_{z} | z) = z + \Sigma_{z} \nabla_{z} \log p(z)$。

在diffusion model中，$x_t\sim \mathcal{N}(\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I)$$x_{t} \sim \mathcal{N}(\sqrt{\bar{\alpha}_{t}}x_{0}, (1-\bar{\alpha}_{t})I)$，所以给定一个$x_t$$x_{t}$，对均值$\sqrt{{\overline{\alpha }}_t}{x}_0$$\sqrt{\bar{\alpha}_{t}} x_{0}$的估计为$x_t+(1-{\overline{\alpha }}_t){\mathrm{\nabla }}_{x_t}\log q(x_t|x_0)$$x_{t} + (1-\bar{\alpha}_{t}) \nabla_{x_{t}} \log q(x_{t}|x_{0})$，根据公式$x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}{ϵ}_t$$x_{t} = \sqrt{\bar{\alpha}_{t}} x_{0} + \sqrt{1-\bar{\alpha}_{t}} \epsilon_{t}$，可以推出${\mathrm{\nabla }}_{x_t}\log q(x_t|x_0)=-\frac{{ϵ}_t}{\sqrt{1-{\overline{\alpha }}_t}}$$\nabla_{x_{t}} \log q(x_{t}|x_{0}) = - \frac{\epsilon_{t}}{\sqrt{1-\bar{\alpha}_{t}}}$。

Awesome Architecture

Cascaded

Cascaded Diffusion Models for High Fidelity Image Generation

MDM

Matryoshka Diffusion Models

所有resolution一起训练，训练时对每个data的所有resolution的加噪时间步要一样，避免信息泄露，同样使用了SimpleDiffusion提出的noise schedule shift。

类似ProgressiveGAN，从低resolution开始训练，之后逐渐加宽UNet增多loss项数去训练更高resolution，训练高resolution时，低resolution网络也会一起训练。

ADM

Diffusion Models Beat GANs on Image Synthesis

AdaGN：将class映射到指定维度，和time embedding相加，通过一个MLP预测$y_s$$y_s$和$y_b$$y_b$，对GroupNorm的结果做affine：（$y_s\cdot GroupNorm(h)+y_b$$y_s \cdot GroupNorm(h) + y_b$）。

super resolution model：将低分辨率图像$x_0^{low}$$x_{0}^{low}$上采样到$x_t^{high}$$x_{t}^{high}$的尺寸，在channel维度上和$x_t^{high}$$x_{t}^{high}$concat在一起，输入到UNet中预测$x_t^{high}$$x_{t}^{high}$的噪声，UNet输入6通道，输出3通道。注意对不同$t$$t$的$x_t^{high}$$x_{t}^{high}$，concat的都是$x_0^{low}$$x_{0}^{low}$，而不是$x_t^{low}$$x_{t}^{low}$。

D3PM

Structured Denoising Diffusion Models in Discrete State-Spaces

1. Discrete Diffusion Model

GGM

Glauber Generative Model: Discrete Diffusion Models via Binary Classification

1. D3PM的前向过程每一步对所有token进行随机加噪，GGM每一步只对某个token进行随机加噪，这样逆向过程每一步只需要预测一个token。

2. 随机一个长度为$T$$T$的index序列$\{i_t{\}}_{t=0}^{T-1}$$\{i_{t}\}_{t=0}^{T-1}$，加噪时，让$x_{t+1}$$x_{t+1}$的除$i_t$$i_{t}$位置之外的token保持不变，$i_t$$i_{t}$位置的token进行随机加噪（也有可能保持不变）。训练和采样时使用相同的index序列。

ImageBART

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

1. VQGAN + multinomial diffusion

2. Transformer Encoder： $x_{t-1}$$x_{t-1}$，Transformer Decoder： AR shifted prediting $x_t$$x_t$

VQ-Diffusion

Vector Quantized Diffusion Model for Text-to-Image Synthesis

1. VQVAE + multinomial diffusion

2. Transformer Blocks：input $x_{t-1}$$x_{t-1}$，cross-attention with text，NAR prediting $x_t$$x_t$

LDM

High-Resolution Image Synthesis with Latent Diffusion Models

AutoEncoder：$H\times W\times 3\to \frac{H}{f}\times \frac{W}{f}\times C$$H \times W \times 3 \rightarrow \frac{H}{f} \times \frac{W}{f} \times C$， $f=2^m$$f=2^m$ ，训练一个AutoEncoder对图像进行适当降维。

固定AutoEncoder，使用UNet训练DDPM建模数据降维后的latent space。好处：减少计算量；不失图像的结构性，仍能发挥UNet的归纳偏置；学到了一个可以更进一步利用的latent space。$m$$m$一般取4或8最好。

slight regularization：KL or VQ，避免 high-variance latent spaces。

为UNet注入cross-attention层，放在self-attention之后。Q: $x_t$$x_t$ feature map KV: conditions

Wuerstchen

StableCascade

Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models

three-stages to reduce computational demands

1. stage A：训练4倍降采样率的VQGAN,$1024\to 256$$1024 \rightarrow 256$。

2. Semantic Compressor：将图像从1024 resize到768，训练一个网络将其压缩到$16\times 24\times 24$$16 \times 24 \times 24$。

3. stage B：diffusion建模stage A中图像quantize之前的embedding，以图像经过semantic compressor的输出为条件（Wuerstchen还以text为额外条件），相当于self-condition。

4. stage C：diffusion建模图像经过semantic compressor的输出，以text为条件。

5. 生成时$C\to B\to A$$C \rightarrow B \rightarrow A$。

BinaryLDM

Binary Latent Diffusion

和LDM一样的思路，只是latent是二值化的。

利用VQ的思路，使用Bernoulli采样取代VQ的最近邻，训练一个隐变量二值化的binary AutoEncoder：$x\in R^{h\times w\times 3}$$x\in R^{h \times w \times 3}$，$y=Sigmoid(E(x))$$y=Sigmoid(E(x))$，$z=Bernoulli(y)$$z=Bernoulli(y)$，$\hat{z}=sg(z)+y-sg(y)$$\hat{z}=sg(z)+y-sg(y)$，$\hat{x}=D(\hat{z})$$\hat{x}=D(\hat{z})$。

推导Bernoulli Diffusion Process，使用DPM建模$z$$z$的分布。

SimpleDiffusion

Simple Diffusion: End-to-end diffusion for high resolution images

1. 现有的高分辨率扩散模型有两种，一种是StableDiffusion的降维法，一种是coarse-to-fine的cascade super resolution法。SimpleDiffusion采用以下几个方法解决高分辨率扩散模型的像素空间的直接训练问题。

2. 改变noise schedule：一个现象是，高分辨率图像$x_0$$x_{0}$的加噪图像$x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ$$x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$相比低分辨率的更具有辨识度，这是因为高分辨率图像像素多，表达一个细节的像素数量很多，相邻像素提供了较大的信息冗余度，参考On the Importance of Noise Scheduling for Diffusion Models。这就导致高分辨率图像的扩散模型的扩散过程前轻后重，生成时也必须在较早且较短的时间内构建图像的大体结构，导致训练效果很差。解决方法是使用一个基准分辨率64x64，并为其设置一个实验中表现较好的经验$SNR$$SNR$函数，然后根据目标分辨率计算相应的$SNR$$SNR$函数，得到shifted noise schedule。

3. 多尺度训练：高分辨率扩散模型的像素空间的直接训练的一个难点在于图像的高频信息（物体边缘等）较难建模，训练loss主要由这部分支配，所以本论文提出多尺度训练loss：$L_{\theta }^{d\times d}(x)=\frac{1}{d^2}{\mathbb{E}}_{ϵ,t}||D^{d\times d}[ϵ]-D^{d\times d}[{ϵ}_{\theta }(x_t,t)]|{|}_2^2$$L_{\theta}^{d\times d}(x)=\frac{1}{d^{2}}\mathbb{E}_{\epsilon,t}||D^{d\times d}[\epsilon]-D^{d\times d}[\epsilon_{\theta}(x_{t},t)]||^{2}_{2}$，其中$D^{d\times d}[\cdot ]$$D^{d\times d}[\cdot]$代表下采样到$d\times d$$d\times d$分辨率，这是因为下采样是线性算子，$D^{d\times d}[{ϵ}_{\theta }]$$D^{d\times d}[\epsilon_{\theta}]$可以被看成是一个$d\times d$$d\times d$的扩散模型。最终优化：$\sum _{s\in \{32,64,128,\cdots ,d\}}\frac{1}{s}{L}_{\theta }^{s\times s}(x)$$\sum_{s\in \{32,64,128,\cdots,d\}}\frac{1}{s}L_{\theta}^{s\times s}(x)$。

4. 为了解决显存和计算问题，在低分辨率的feature map上增加网络深度，即block数量，本论文选择16x16，并且在整个模型最前面加一个下采样层，在最后面加一个上采样层，避免在最高分辨率下做计算。

5. 只在低分辨率feature map上加dropout。

Scaling-Law-1

On the Scalability of Diffusion-based Text-to-Image Generation

1. For model scaling, we find the location and amount of cross attention distinguishes the performance of existing UNet designs. And increasing the transformer blocks is more parameter-efficient for improving text-image alignment than increasing channel numbers.

2. On the data scaling side, we show the quality and diversity of the training set matters more than simply dataset size. Increasing caption density and diversity improves text-image alignment performance and the learning efficiency.

Scaling-Law-2

Bigger is not Always Better: Scaling Properties of Latent Diffusion Models

1. When operating under a given inference budget, smaller models frequently outperform their larger equivalents in generating high-quality results.

Scaling-Law-3

Computational Tradeoffs in Image Synthesis Diffusion, Masked-Token, and Next-Token Prediction

1. We recommend diffusion for applications targeting image quality and low latency; and next-token prediction when prompt following or throughput is more important.

RDM

Relay Diffusion: Unifying diffusion process across resolutions for image synthesis

1. cascaded models perform better than end-to-end models under a fair setting，RDM也是用cascaded模式，但是与传统cascaded models不同的是，RDM是在时间步上cascade，这降低了训练和采样的步数。

2. 高低分辨率都使用EDM formulation，即$x_t=x+\sigma ϵ$$x_{t} = x + \sigma \epsilon$。

3. 想要在$t$$t$处relay，高低分辨率分别训练的diffusion model在$t$$t$处的$x_t$$x_{t}$的SNR需要匹配，但是the same noise level on a higher resolution results in a higher Signal-to-Noise Ratio in the frequency domain，因此提出和shifted noise schedule不同的方法Block Noise，在$64\times 64$$64 \times 64$采样一个噪声，然后使用Block Noise上采样到$256\times 256$$256 \times 256$​，这两个噪声加到各自分辨率的图像上后SNR就相同了。

4. 低分辨率生成的$x_t$$x_{t}$和高分辨率的$x_t$$x_{t}$还是有gap的，因此高分辨率使用blurring diffusion建模，前向过程不仅加噪，还会对$x$$x$进行blur，这样就相当于把低分辨率生成的$x_t$$x_{t}$的上采样看成是高分辨率图像被blur后的加噪结果。

5. 注意，blurring diffusion建模时使用Block Noise加噪（不是直接采样$256\times 256$$256 \times 256$的噪声），这样$x_t$$x_{t}$上采样后就能直接输入blurring diffusion进行生成。

U-ViT

All are Worth Words: A ViT Backbone for Diffusion Models

1. ViT in Pixel Space

Diffusion-RWKV

Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models

1. RWKV brought improvements for standard RNN architecture, which is computed in parallel during training while inference like RNN. It involves enhancing the linear attention mechanism and designing the receptance weight key value (RWKV) mechanism.

DiT

Scalable Diffusion Models with Transformers

1. ViT in Latent Space

2. adaLN：LayerNorm中不学习scale和shift，而是额外使用一个MLP（每个block都有）根据timestep和condition预测一个scale和shift。

3. adaLN-Zero：在skip-connection前再乘一个预测出来的scale，初始化MLP使这个scale的输出都为$0$$0$，这样整个DiT Block就被初始化为indentity function，有利于训练。

U-DiT

U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers

1. When the encoder processes the input image by downsampling the image as stage-level amounts, the decoder scales up the encoded image from the most compressed stage to input size. At each encoder stage transition, spatial downsampling by the factor of $2$$2$ is performed while the feature dimension is doubled as well. Skip connections are provided at each stage transition. The skipped feature is concatenated and fused with the upsampled output from the previous decoder stage, replenishing information loss to decoders brought by feature downsampling.

2. 类似ToDo，使用token downsampling减少计算量，但并不丢失信息：将$N\times N$$N \times N$的feature降采样（使用depthwise convolution）为$4$$4$个$\frac{N}{2}\times \frac{N}{2}$$\frac{N}{2} \times \frac{N}{2}$的feature，每个feature独自做self-attention，$4$$4$个self-attention结果拼回$N\times N$$N \times N$，比如$4$$4$个self-attention结果的第一个元素拼成一个$2\times 2$$2\times 2$​的grid作为最终结果的第一个元素，总计算量减少$\frac{1}{4}$$\frac{1}{4}$，但并没有信息丢失。Unlike U-Net downsampling, we are not reducing or increasing the number of elements in the feature during the downsampling process. The substitution of downsampled self-attention to full-scale self-attention brings slight improvement in the FID metric despite a significant reduction in FLOPs.

Switch-DiT

Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts

1. 引入MoE，每个block都使用timestep-based gating network预测一个概率分布，取TopK，这样可以做到参数隔离，缓解不同时间步冲突的问题。

SiT

SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers

1. design space + DiT

HDiT

Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers

1. 传统Transformer每一个block计算量都是一样的，这里把UNet思想用在Transformer上，中间层减少token数量，降低运算量。

2. 同时借鉴SimpleDiffusion，高分辨率少做计算，低分辨率增长增宽。

3. 这样可以在pixel层面的计算复杂度只随着图像分辨率提升线性增长。

DiMR

Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models

1. U-ViT

2. Transformer在计算量和效果之间存在tradeoff，patch size小时，token length长，计算量大，但效果好，patch size大时，token length短，计算量小，但效果差。

3. feature cascade：分为$R$$R$个branch，所有branch都是用同一个加噪后的$x_t$$x_{t}$。对于第$r$$r$个branch，使用$2^{R-r}\times 2^{R-r}$$2^{R-r} \times 2^{R-r}$的卷积降低输入$x_t$$x_{t}$的分辨率，该branch的输出上采样为原来的两倍，concat在下一branch的输入上。在每个branch的输出上都计算一个diffusion loss，目标为相同的$ϵ$$\epsilon$的$r$$r$​倍的average pooling下采样。

4. 在低分辨率上使用U-ViT架构，在高分辨率使用ConvNeXt节省计算量。

Flag-DiT

Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers

1. Flag-DiT substitutes all LayerNorm with RMSNorm to improve training stability. Moreover, it incorporates key-query normalization (KQ-Norm) before key-query dot product attention computation. The introduction of KQ-Norm aims to prevent loss divergence by eliminating extremely large values within attention logits.

2. We introduce learnable special tokens including the [nextline] and [nextframe] tokens to transform training samples with different scales and durations into a unified one-dimensional sequence. We add [PAD] tokens to transform 1-D sequences into the same length for better parallelism. 不同模态的数据都转换为一个1D序列统一建模。

3. 有text时，self-attention和cross-attention并列。

FiT

FiT: Flexible Vision Transformer for Diffusion Model

1. ViT in Latent Space

2. 不做crop，保持长宽比，resize图像使其满足$HW\le {256}^2$$HW \le 256^{2}$，VAE下采样8倍，patchify size=2，所以token长度最多为$(\frac{\frac{256}{8}}{2}{)}^2=256$$(\frac{\frac{256}{8}}{2})^{2}=256$​，不够256的pad到256，MHSA时mask掉pad token，只在unmask token上计算loss。

3. 使用2D-RoPE位置编码方法，利用其外推性可以生成任意分辨率和长宽比的图像。

VisionLLaMA

VisionLLaMA: A Unified LLaMA Interface for Vision Tasks

1. ViT in Latent Space

2. a vision transformer architecture similar to LLaMA to reduce the architectural differences between language and vision.

DiG

DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention

1. ViT in Latent Space

2. DiT models have faced challenges with scalability and quadratic complexity efficiency. We leverage the long sequence modeling capability of Gated Linear Attention (GLA) Transformers, expanding its applicability to diffusion models and offering superior efficiency and effectiveness.

Mamba

DiS

Scalable Diffusion Models with State Space Backbone

DiffuSSM

Diffusion Models Without Attention

ZigMa

ZigMa: Zigzag Mamba Diffusion Model

DiM

DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis

DiM

Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation

Dimba

Dimba: Transformer-Mamba Diffusion Models

Others

Infinite-Diff

Infinite-Diff: Infinite Resolution Diffusion with Subsampled Mollified States

1. Infinite-Diff is a generative diffusion model defined in an infinite dimensional Hilbert space, which can model infinite resolution data. By training on randomly sampled subsets of coordinates and denoising content only at those locations, we learn a continuous function for arbitrary resolution sampling.

INFD

Image Neural Field Diffusion Models

1. Neural field is also known as Implicit Neural Representations (INR), which represents signals as coordinate-based neural networks.

2. 提出Image Neural Field Autoencoder，目的是有一个隐空间分布可以建模并采样。

3. 类似Diff-AE和PDAE，使用diffusion model建模隐空间分布。

CAN

Condition-Aware Neural Network for Controlled Image Generation

1. 传统的条件模型中，所有条件共用相同的处理条件的static network，这限制了网络的建模能力。一种解决方案是每个条件使用一个expert model，但消耗极大。因此学习一个生成网络，根据条件动态生成处理条件的网络参数。introduces a condition-aware weight generation module that generates conditional weight for convolution/linear layers based on the input condition.

2. Making depthwise convolution layers, the patch embedding layer, and the output projection layers condition-aware brings a significant performance boost.

3. CAN consistently delivers significant improvements for diffusion transformer models, including DiT and UViT, on class conditional image generation on ImageNet and text-to-image generation on COCO.

Effectiveness and Efficiency Enhancement

hybrid generative model

D2C

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation

diffusion建模autoencoder的latent distribution。

DDGAN

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

1. 解决原始扩散模型不可训练的问题，参考theory第6章。

2. 步长较大时，$q(x_{t-1}|x_t)$$q(x_{t-1} | x_{t})$就不再是高斯分布，变得更复杂和多峰，所以可以使用对抗学习拟合$p_{\theta }(x_{t-1}|x_t)$$p_{\theta}(x_{t-1} | x_{t})$，即$\underset{\theta }{min}\sum _{t\ge 1}{E}_{q(x_t)}[D_{\text{adv}}(q(x_{t-1}|x_t)\parallel p_{\theta }(x_{t-1}|x_t))]$$\min_{\theta} \sum_{t \ge 1} E_{q(x_{t})}[D_{\text{adv}}(q(x_{t-1}|x_{t}) \parallel p_{\theta}(x_{t-1} | x_{t}))]$，可以转化为同时学习判别器$\underset{\varphi }{min}\sum _{t\ge 1}{E}_{q(x_t)}\{E_{q(x_{t-1}|x_t)}[-\log D_{\varphi }(x_{t-1},x_t,t)]+E_{p_{\theta }(x_{t-1}|x_t)}[-\log (1-D_{\varphi }(x_{t-1},x_t,t))]\}$$\min_{\phi} \sum_{t \ge 1} E_{q(x_{t})} \{E_{q(x_{t-1} | x_{t})}[- \log D_{\phi}(x_{t-1}, x_{t}, t)] + E_{p_{\theta}(x_{t-1} | x_{t})}[- \log (1 - D_{\phi}(x_{t-1}, x_{t}, t))]\}$和生成器$\underset{\theta }{max}\sum _{t\ge 1}{E}_{q(x_t)}{E}_{p_{\theta }(x_{t-1}|x_t)}[\log D_{\varphi }(x_{t-1},x_t,t)]$$\max_{\theta} \sum_{t \ge 1} E_{q(x_{t})} E_{p_{\theta}(x_{t-1} | x_{t})} [\log D_{\phi}(x_{t-1}, x_{t}, t) ]$。

3. $q(x_t)q(x_{t-1}|x_t)=q(x_{t-1},x_t)=\int d{x}_{0}q(x_0,x_{t-1},x_t)=\int d{x}_{0}q(x_0)q(x_{t-1}|x_0)q(x_t|x_{t-1},x_0)=\int d{x}_{0}q(x_0)q(x_{t-1}|x_0)q(x_t|x_{t-1})$$q(x_{t})q(x_{t-1}|x_{t})=q(x_{t-1},x_{t})=\int dx_{0} q(x_{0}, x_{t-1}, x_{t}) = \int dx_{0} q(x_{0})q(x_{t-1} | x_{0})q(x_{t} | x_{t-1}, x_{0}) = \int dx_{0} q(x_{0})q(x_{t-1} | x_{0})q(x_{t} | x_{t-1})$，所以先从$x_0$$x_{0}$加噪到$x_{t-1}$$x_{t-1}$，再从$x_{t-1}$$x_{t-1}$加噪到$x_t$$x_{t}$。

4. $p_{\theta }(x_{t-1}|x_t)=\int p_{\theta }(x_0|x_t)q(x_{t-1}|x_t,x_0)d{x}_0=\int p(z)q(x_{t-1}|x_t,x_0=G_{\theta }(z,x_t,t))dz$$p_{\theta}(x_{t-1} | x_{t}) = \int p_{\theta}(x_{0} | x_{t}) q(x_{t-1} | x_{t}, x_{0}) dx_{0} = \int p(z) q(x_{t-1} | x_{t}, x_{0} = G_{\theta}(z, x_{t}, t)) dz$，所以使用生成器根据$x_{t-1}$$x_{t-1}$生成$x_0$$x_{0}$，再使用$q(x_{t-1}|x_t,x_0)$$q(x_{t-1} | x_{t}, x_{0})$采样一个$x_{t-1}$$x_{t-1}$。

5. 使用判别器判别$x_{t-1}$$x_{t-1}$的真假。Note that for different $t$$t$, $x_t$$x_{t}$ has different levels of perturbation, and hence using a single network to predict $x_{t-1}$$x_{t-1}$ directly at different $t$$t$ may be difficult. However, in our case the generator only needs to predict unperturbed $x_0$$x_{0}$ and then add back perturbation using $q(x_{t-1}|x_t,x_0)$$q(x_{t-1} | x_{t}, x_{0})$.

6. DDPMs的逆向过程也可以被解释为$p_{\theta }(x_{t-1}|x_t)=q(x_{t-1}|x_t,x_0=\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}})$$p_{\theta}(x_{t-1} | x_{t}) = q(x_{t-1} | x_{t}, x_{0} = \frac{x_{t} - \sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}(x_{t},t)}{\sqrt{\bar{\alpha}_{t}}})$，它虽然一直是一个高斯分布，但当步长较大步数较少时效果较差，其和DDGANs的区别在于预测的$x_0$$x_{0}$是否为确定性的，由于步长很小时$p_{\theta }(x_{t-1}|x_t)$$p_{\theta}(x_{t-1} | x_{t})$是高斯分布，所以其预测的$x_0$$x_{0}$是确定性的，而DDGANs步长很大，预测的$x_0$$x_{0}$是具有随机性的，因此$p_{\theta }(x_{t-1}|x_t)$$p_{\theta}(x_{t-1} | x_{t})$就是多峰的。

7. GANs are known to suffer from training instability and mode collapse, and some possible reasons include the difficulty of directly generating samples from a complex distribution in one-shot, and the overfitting issue when the discriminator only looks at clean samples. Our model breaks the generation process into several conditional denoising diffusion steps in which each step is relatively simple to model, due to the strong conditioning on $x_t$$x_{t}$. Moreover, the diffusion process smoothens the data distribution, making the discriminator less likely to overfit.

DiffuseVAE

DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents

FM-Boosting

Boosting Latent Diffusion with Flow Matching

1. LDM的推理速度随着图像分辨率提高而平方增长。

2. 使用Flow Matching在上采样的低分辨率的latent和高分辨率的latent之间建模。使用低分辨率LDM进行生成，使用Flow Matching转换到高分辨率。

3. 一般的Flow Matching是在数据分布和高斯分布之间建模的，这里是在数据对之间建模，所以叫Coupling Flow Matching。

PDM

Ultra-High-Resolution Image Synthesis with Pyramid Diffusion Model

1. 类似FM-Boosting的思路逐步生成高分辨率的图像。

refinement of network architectures

LDM

High-Resolution Image Synthesis with Latent Diffusion Models

LiteVAE

LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models

1. We leverage the 2D discrete wavelet transform to enhance scalability and computational efficiency over standard variational autoencoders (VAEs) with no sacrifice in output quality.

Wuerstchen

`Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models

SimpleDiffusion

Simple Diffusion: End-to-end diffusion for high resolution images

BK-SDM

BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion

1. compact UNet：fewer blocks in the down and up stages (Base), further removal of the entire mid-stage (Small), further removal of the innermost stages (Tiny).

2. distillation-based retraining：除了diffusion loss，还可以使用训练好的大网络的StableDiffusion进行output-level distillation（相同输入得到的输出之间的MSE loss）和feature-level distillation（相同输入得到的网络feature之间的MSE loss）。

KOALA

KOALA: Fast and Memory-Efficient Latent Diffusion Models via Self-Attention Distillation

1. 和BK-SDM做法一样。

2. 进一步改进了feature-level distillation，测试了使用不同模块输出的feature进行distillation时的效果，发现对self-attention输出的feature进行distillation时效果最好，而且decoder early blocks位置的self-attention输出的feature效果最好。

HDiT

Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers

SiT

SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers

DiG

DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention

SnapFusion

SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds

MobileDiffusion

MobileDiffusion: Subsecond Text-to-Image Generation on Mobile Devices

lightweight model architecture + DiffusionGAN + distillation

Spiking Network

Spiking-Diffusion Vector Quantized Discrete Diffusion Model with Spiking Neural Networks

Spiking Denoising Diffusion Probabilistic Models

Fully Spiking Denoising Diffusion Implicit Models

SDiT: Spiking Diffusion Model with Transformer

NAS

AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration

Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search

DDSM

Denoising Diffusion Step-aware Models

不同step的重要性是不同的，没必要每一步都使用大模型。

slimmable network： a neural network that can be executed at arbitrary model sizes.

搜索最优的采样策略，不同步使用不同size的模型，减少计算量。

ScaleLong

ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection

feature norm的角度理论分析UNet的long skip connection coefficient的影响。

Q-DM

Q-DM: An Efficient Low-bit Quantized Diffusion Model

DiffuSSM

Diffusion Models Without Attention

EDM-2

Analyzing and Improving the Training Dynamics of Diffusion Models

1. We update all of the operations (e.g., convolutions, activations, concatenation, summation) to maintain magnitudes on expectation.

ReDistill

ReDistill: Residual Encoded Distillation for Peak Memory Reduction

1. Reducing peak memory, which is the maximum memory consumed during the execution of a neural network, is critical to deploy neural networks on edge devices with limited memory budget. We propose residual encoded distillation (ReDistill) for peak memory reduction in a teacher-student framework, in which a student network with less memory is derived from the teacher network using aggressive pooling.

Quantum

Quantum Denoising Diffusion Models

Quantum Generative Diffusion Model

Towards Efficient Quantum Hybrid Diffusion Models

量子计算

pre-trained network compression

quantization

PTQ

Post-training Quantization on Diffusion Models

Q-Diffusion

Q-Diffusion: Quantizing Diffusion Models

SD-PTQ

Post-training Quantization with Progressive Calibration and Activation Relaxing for Text-to-Image Diffusion Models

LDM-PTQ

Efficient Quantization Strategies for Latent Diffusion Models

TerDiT

TerDiT: Ternary Diffusion Models with Transformers

PTQ4DiT

PTQ4DiT: Post-training Quantization for Diffusion Transformers

HQ-DiT

HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization

StableQ

StableQ: Enhancing Data-Scarce Quantization with Text-to-Image Data

BinaryDM

BinaryDM: Towards Accurate Binarization of Diffusion Model

APQ-DM

Towards Accurate Post-training Quantization for Diffusion Models

TFMQ-DM

TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models

EfficientDM

EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models

EDA-DM

Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models

Memory-Efficient

Memory-Efficient Personalization using Quantized Diffusion Model

fine-tune quantized diffusion model

MixDQ

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization

COMQ

COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization

QNCD

QNCD: Quantization Noise Correction for Diffusion Models

network pruning

Diff-Pruning

Structural Pruning for Diffusion Models

Lottery-Ticket-to-DDPM

Successfully Applying Lottery Ticket Hypothesis to Diffusion Model

LAPTOP-Diff

LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models

LD-Pruner

LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights

LayerMerge

LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging

Token Reduction

ToMe

Token Merging: Your ViT But Faster

Token Merging for Fast Stable Diffusion

AT-EDM

Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models

1. 根据attention map识别过剩的token进行merge。

ToDo

ToDo: Token Downsampling for Efficient Generation of High-Resolution Images

1. 类似PixArt-$\mathrm{\Sigma }$$\Sigma$的KV compression，但是training-free的。

2. Tokens in close spatial proximity exhibit higher similarity, thus providing a basis for merging without the extensive com putation of pairwise similarities.

3. We employ a downsampling function using the Nearest-Neighbor algorithm to the keys and values of the attention mechanism while preserving the original queries.

enhancement of training

novel model family

iDDPM

Improved Denoising Diffusion Probabilistic Models

1. They achieve similar sample quality using either ${\sigma }_t^2={\beta }_t$$\sigma_{t}^{2}=\beta_{t}$ or ${\sigma }_t^2={\tilde{\beta }}_t$$\sigma_{t}^{2}=\tilde{\beta}_{t}$ which are the upper and lower bounds on the variance given by $q(x_0)$$q(x_{0})$ being either isotropic Gaussian noise or a delta function (${\tilde{\beta }}_0=0$$\tilde{\beta}_{0}=0$), respectively. We choose to parameterize the variance as an interpolation between ${\beta }_t$$\beta_{t}$ and ${\tilde{\beta }}_t$$\tilde{\beta}_{t}$ in the log domain. ${\mathrm{\Sigma }}_{\theta }(x_t,t)=\exp (v\log {\beta }_t+(1-v)\log {\tilde{\beta }}_t)$$\Sigma_{\theta}(x_{t}, t) = \exp( v \log \beta_{t} + (1-v) \log \tilde{\beta}_{t})$. We did not apply any constraints on model output $v$$v$, theoretically allowing the model to predict variances outside of the interpolated range. However, we did not observe the network doing this in practice, suggesting that the bounds for ${\mathrm{\Sigma }}_{\theta }(x_t,t)$$\Sigma_{\theta}(x_{t}, t)$ are indeed expressive enough.

2. 除了diffusion loss，额外优化$0.001\cdot {\mathcal{L}}_{\text{vlb}}$$0.001 \cdot \mathcal{L}_{\text{vlb}}$，we also apply a stop-gradient to the ${\mu }_{\theta }(x_t,t)$$\mu_{\theta}(x_{t}, t)$ output for the ${\mathcal{L}}_{\text{vlb}}$$\mathcal{L}_{\text{vlb}}$​ term.

3. sampling $t$$t$ uniformly causes unnecessary noise in the ${\mathcal{L}}_{\text{vlb}}$$\mathcal{L}_{\text{vlb}}$，根据历史${\mathcal{L}}_t$$\mathcal{L}_{t}$在所有${\mathcal{L}}_t$$\mathcal{L}_{t}$的占比确定$t$$t$的采样概率。

PD

Progressive Distillation for Fast Sampling of Diffusion Models

1. v-prediction，在理论上和$ϵ$$\epsilon$-prediction是等价的，因为两者可以相互转换。

FM

Flow Matching for Generative Modeling

1. 基于Continuous Normalizing Flows（Neural ODE），CNFs在训练时是先使用模型对数据样本进行转化（ODE simulations），计算转化后的样本与标准高斯分布之间的KL散度，优化模型；flow matching是simulation-free的，因为ODE路径已经提前定义好了。

2. diffusion model和score based model对${\mathrm{\nabla }}_{x_t}\log p_t(x_t)$$\nabla_{x_{t}} \log p_{t}(x_{t})$进行拟合，flow matching对$\frac{d}{dt}{x}_t$$\frac{d}{dt} x_{t}$进行拟合，如果和diffusion model或score based model使用相同的扩散核，那么flow matching在理论上和diffusion model和score based model是等价的（因为$v_{\theta }$$v_{\theta}$和${ϵ}_{\theta }$$\epsilon_{\theta}$或$s_{\theta }$$s_{\theta}$​​都可以相互转换，类似PD的v-prediction）。we find this training alternative to be more stable and robust in our experiments to existing score matching approaches.

RectFlow

Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

1. 可以在任意两个分布之间进行建模。

2. 随机从两个分布中的选择数据$X_0$$X_{0}$和$X_1$$X_{1}$（不需要成对），使用最简单的插值法$X_t=t{X}_1+(1-t)X_0$$X_{t} = t X_{1} + (1-t)X_{0}$，此时$\frac{d}{dt}{X}_t=X_1-X_0$$\frac{d}{dt} X_{t} = X_{1} - X_{0}$，所以损失函数为${\int }_0^{1}E\parallel (X_1-X_0)-v_{\theta }(X_t,t){\parallel }^{2}dt$$\int_{0}^{1} E \parallel (X_{1} - X_{0}) - v_{\theta}(X_{t},t) \parallel^{2} dt$​。

3. 每次训练完成后进行采样，得到当前flow下的coupling数据，之后用这些couping数据再进行训练，以此循环对轨迹进行修正，会得到直的没有交叉点的flow。

EDM

Elucidating the Design Space of Diffusion-Based Generative Models

1. $x_t=x_0+ϵ$$x_{t} = x_{0} + \epsilon$，$ϵ\sim \mathcal{N}(0,{\sigma }_t^{2}I)$$\epsilon \sim \mathcal{N}(0, \sigma_{t}^{2} I)$，$D_{\theta }(x_t,{\sigma }_t)\to x_0$$D_{\theta}(x_{t}, \sigma_{t}) \rightarrow x_{0}$。

2. preconditioning：As the input $x_t$$x_{t}$ is a combination of clean signal and noise, its magnitude varies immensely depending on noise level ${\sigma }_t$$\sigma_{t}$. For this reason, the common practice is to not represent $D_{\theta }$$D_{\theta}$ as a neural network directly, but instead train a different network $F_{\theta }$$F_{\theta}$ from which $D_{\theta }$$D_{\theta}$ can be derived. VE trains $F_{\theta }$$F_{\theta}$ to predict $ϵ$$\epsilon$ scaled to unit variance, from which the signal is then reconstructed via $D_{\theta }(x_t,{\sigma }_t)=x_t-\sigma F_{\theta }(x_t,{\sigma }_t)$$D_{\theta}(x_{t}, \sigma_{t}) = x_{t} - \sigma F_{\theta}(x_{t}, \sigma_{t})$. This has the drawback that at large $\sigma$$\sigma$, the network needs to fine-tune its output carefully to cancel out the existing noise $ϵ$$\epsilon$ exactly and give the output at the correct scale. Note that any errors made by the network are amplified by a factor of ${\sigma }_t$$\sigma_{t}$. In this situation, it would seem much easier to predict the expected output $D_{\theta }(x_t,{\sigma }_t)$$D_{\theta}(x_{t}, \sigma_{t})$ directly. To this end, we propose to precondition the neural network with a $\sigma$$\sigma$-dependent skip connection that allows it to estimate either $x_0$$x_{0}$ or $ϵ$$\epsilon$, or something in between. $D_{\theta }(x_t,{\sigma }_t)=c_{\text{skip}}({\sigma }_t)x_t+c_{\text{out}}({\sigma }_t)F_{\theta }(c_{\text{in}}({\sigma }_t)x_t,{\sigma }_t)\to x_0$$D_{\theta}(x_{t}, \sigma_{t}) = c_{\text{skip}}(\sigma_{t})x_{t} + c_{\text{out}}(\sigma_{t}) F_{\theta}(c_{\text{in}}(\sigma_{t})x_{t}, \sigma_{t}) \rightarrow x_{0}$，$F_{\theta }(c_{\text{in}}({\sigma }_t)x_t,{\sigma }_t)\to \frac{1}{c_{\text{out}}({\sigma }_t)}(x_0-c_{\text{skip}}({\sigma }_t)x_t)$$F_{\theta}(c_{\text{in}}(\sigma_{t})x_{t}, \sigma_{t}) \rightarrow \frac{1}{c_{\text{out}}(\sigma_{t})}(x_{0} - c_{\text{skip}}(\sigma_{t})x_{t})$，we choose $c_{\text{in}}$$c_{\text{in}}$ and $c_{\text{out}}$$c_{\text{out}}$ to make the network inputs and training targets have unit variance, and $c_{\text{skip}}$$c_{\text{skip}}$ to make the amplifying errors in $F_{\theta }$$F_{\theta}$ as little as possible. Other diffusion models always have $c_{\text{skip}}=1$$c_{\text{skip}}=1$.

3. augmentation: To prevent potential overfitting that often plagues diffusion models with smaller datasets, we apply various geometric transformations to a training image prior to adding noise. To prevent the augmentations from leaking to the generated images, we provide the augmentation parameters as a conditioning input to $F_{\theta }$$F_{\theta}$; during inference we set the them to zero to guarantee that only non-augmented images are generated. (macro conditioning as augmentation)

VDM

Variational Diffusion Models

efficient optimization of the noise schedule jointly with the rest of the model

VDM++

Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation

DiffEnc

DiffEnc: Variational Diffusion with a Learned Encoder

SPD

Image generation with shortest path diffusion

optimal noise schedule

Cosine

Improved Denoising Diffusion Probabilistic Models

1. 与linear schedule不同，cosine schedule直接定义${\overline{\alpha }}_t={\cos }^2(\frac{t+0.008}{1.008}\cdot \frac{\pi }{2})$$\bar{\alpha}_{t} = \cos^{2} (\frac{t+0.008}{1.008}\cdot \frac{\pi}{2})$，再计算出${\beta }_t$$\beta_{t}$。

FixFlaw

Common Diffusion Noise Schedules and Sample Steps are Flawed

1. StableDiffusion使用的noise schedule最后一步加噪公式为$z_T=0.068265z_0+0.997667ϵ$$z_{T} = 0.068265 z_{0} + 0.997667 \epsilon$，并不是标准高斯分布，$z_T$$z_{T}$仍然包含一些信息，其均值不为$0$$0$，黑色（-1）的图像均值为负，白色（1）的图像均值为正，在生成时使用随机的$z_T$$z_{T}$，均值为$0$$0$，所以生成的都是中等亮度的图像。随机一个$ϵ$$\epsilon$，使用其给某个图像加噪到$z_T$$z_{T}$，从$z_T$$z_{T}$开始生成的结果和从$ϵ$$\epsilon$开始生成的结果也不同，参考Magic-Fixup。

2. 为了解决这个问题，需要保证zero terminal SNR，即$\overline{{\alpha }_T}=0$$\bar{\alpha_{T}}=0$，使用rescale schedule方法对现有的noise schedule进行修正，做法是保持$\sqrt{{\overline{\alpha }}_1}$$\sqrt{\bar{\alpha}_{1}}$不变，修正$\sqrt{\overline{{\alpha }_T}}=0$$\sqrt{\bar{\alpha_{T}}}=0$，之后对$2,\cdots ,T$$2, \cdots, T$的$\sqrt{\overline{{\alpha }_t}}$$\sqrt{\bar{\alpha_{t}}}$进行rescale，之后再重新训练模型。

3. 使用了zero terminal SNR的noise schedule后，$T$$T$时刻的denoising loss就没有意义了，所以建议使用PD提出的v-prediction参数法，$v_t=\sqrt{{\overline{\alpha }}_t}ϵ-\sqrt{1-{\overline{\alpha }}_t}{x}_0$$v_{t} = \sqrt{\bar{\alpha}_{t}}\epsilon - \sqrt{1-\bar{\alpha}_{t}}x_{0}$，这样$v_T=-x_0$$v_{T} = -x_{0}$，$v_1=\sqrt{{\overline{\alpha }}_1}ϵ-\sqrt{1-{\overline{\alpha }}_1}{x}_0$$v_{1} = \sqrt{\bar{\alpha}_{1}}\epsilon - \sqrt{1-\bar{\alpha}_{1}}x_{0}$，预测都有了意义。

4. 结合上述两点，可以使用纠正后的noise schedule和v-prediction方法fine-tune已有的StableDiffusion，效果一致。

5. Rescale Classifier-Free Guidance：使用zero terminal SNR的noise schedule后，原有的cfg会变得敏感，导致生成图像过曝，所以对cfg结果进行rescale。

SingDiffusion

Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models

1. 现有的模型不能生成太亮或太暗的图像。

2. 现有的模型由于没有zero terminal SNR，所以本质上是在$(0,1-ϵ]$$(0,1-\epsilon]$上训练的，因此可以微调预训练扩散模型额外训练一步，或者重新训练一个只有最后一步的扩散模型。同FixFlaw一样，由于$T$$T$时刻的denoising loss没有意义，所以这一步的训练使用$x$$x$-predicition。

3. 采样时第一步从高斯分布开始，采样出一个$x_{1-ϵ}$$x_{1-\epsilon}$，之后和原来一样。

optimal loss weighting

P2-weighting

Perception Prioritized Training of Diffusion Models

Debias

Debias the Training of Diffusion Models

类似P2-weighting。

${\hat{x}}_0=\frac{1}{\sqrt{{\overline{\alpha }}_t}}{x}_t-\frac{\sqrt{1-{\overline{\alpha }}_t}}{\sqrt{{\overline{\alpha }}_t}}{ϵ}_{\theta }(x_t,t)=\frac{1}{\sqrt{{\overline{\alpha }}_t}}(\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ)-\frac{\sqrt{1-{\overline{\alpha }}_t}}{\sqrt{{\overline{\alpha }}_t}}{ϵ}_{\theta }(x_t,t)=x_0+\frac{\sqrt{1-{\overline{\alpha }}_t}}{\sqrt{{\overline{\alpha }}_t}}(ϵ-{ϵ}_{\theta }(x_t,t))=x_0+\frac{1}{\sqrt{SNR(t)}}(ϵ-{ϵ}_{\theta }(x_t,t))$$\hat{x}_{0} = \frac{1}{\sqrt{\bar{\alpha}_{t}}}x_{t} - \frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t}, t) = \frac{1}{\sqrt{\bar{\alpha}_{t}}} (\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon) - \frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t}, t) = x_{0} + \frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}} (\epsilon - \epsilon_{\theta}(x_{t}, t)) = x_{0} + \frac{1}{\sqrt{SNR(t)}} (\epsilon - \epsilon_{\theta}(x_{t}, t))$

使用$\frac{1}{\sqrt{SNR(t)}}$$\frac{1}{\sqrt{SNR(t)}}$作为$\parallel {ϵ}_{\theta }(x_t,t)-ϵ{\parallel }_2^2$$\| \epsilon_{\theta}(x_{t},t) - \epsilon \|^{2}_{2}$的weight，使得${\hat{x}}_0$$\hat{x}_{0}$更接近$x_0$$x_{0}$。

SpeeD

A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training

1. Our key findings are: i) Time steps can be empirically divided into acceleration, deceleration, and convergencen areas based on the process increment. ii) These time steps are imbalanced, with many concentrated in the convergence area. iii) The concentrated steps provide limited benefits for diffusion training.

2. We design an asymmetric time step sampling strategy that reduces the frequency of time steps from the convergence area while increasing the sampling probability for time steps from other areas.

Multi-Task Learning

Multi-Architecture Multi-Expert Diffusion Models

Addressing Negative Transfer in Diffusion Models

eDiffi Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Not All Steps are Equal: Efficient Generation with Progressive Diffusion Models

Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture

1. diffusion model需要处理识别所有level的noise这一特点导致了其模型需要大量参数进行训练。

2. 不同时间步（expert）用不同的特性网络（architecture），降低学习难度，减少模型参数量。

Denoising Task Routing for Diffusion Models

Dual-Output

Dynamic Dual-Output Diffusion Models

Cas-DM

Bring Metric Functions into Diffusion Models

Curriculum Learning

DBCL

Denoising Task Difficulty-based Curriculum for Training Diffusion Models

1. diffusion model不同时间步的学习难度是不同的，将时间步等分为20个区间，在每个区间单独训练一个模型（总共20个），考察它们的收敛速度，在loss和生成质量（混合采样，使用一个正常训练好的扩散模型，采样时只在指定区间使用单独训练的模型）方面，都是时间步越大收敛速度越快。

2. Curriculum Learning：a method of training models in a structured order, starting with easier tasks or examples and gradually increasing difficulty. 所以将时间步分区域后，先从最靠后的区域开始训练，之后依次向靠前的区域训练，每次训练时依然要训练之前区域的时间步，避免遗忘。

3. 收敛更快，生成效果更好。

Transfer Learning

Diff-Tuning

Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting

1. Chain of Forgetting：$t$$t$较小时，diffusion model可以做zero-shot denoising，和数据集无关；$t$$t$较大时，diffusion model的泛化性受数据集影响较大。

2. transfer时，同时使用一些原数据集的数据参与训练。使用原数据集的数据时，diffusion loss系数随着$t$$t$的增大单调递减；使用transfer数据集的数据时，diffusion loss系数随着$t$$t$的增大单调递增。

Frequency Domain

SFUNet

Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models

Masked Diffusion

MaskDM

Masked Diffusion Models are Fast Learners

使用U-ViT架构（Pixel Space），最高90%的mask ratio，比DDPM收敛速度快4倍，且生成效果更好。

MDT

Masked Diffusion Transformer is a Strong Image Synthesizer

使用DiT架构（Latent Space），为了解决训练和推理时mask不同的distribution shift，训练时使用一个side-interpolater补全masked tokens，比DiT收敛速度快3倍，生成效果更好。

MaskDiT

Fast Training of Diffusion Models with Masked Transformers

1. 使用DiT架构（Latent Space），DiT encoder可以scaling up，DiT decoder使用固定的$8$$8$个DiT block。

2. 只根据visible token预测invisible token的score太困难了，所以将diffusion loss拆分，对于visible token使用diffusion loss，对于invisible token使用对应noisy patches的MSE loss（注意是直接预测加噪的invisible patch，不是预测其噪声或者原图），类似MaskDM + MAE。

3. 最高50%的mask ratio，比DiT收敛速度快3倍，达到相同的生成效果。

4. MAE必不可少，如果不加MAE，生成效果降低很多，但MAE的loss的系数如果太大反而会影响生成效果，所以这个系数要精心挑选。Without the MAE reconstruction task, the training easily overfits the local subset of unmasked tokens as it lacks a global understanding of the full image, making the gradient update less informative. 理解辅助生成。

SD-DiT

SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer

1. DiT encoder可以scaling up，DiT decoder使用固定的$8$$8$个DiT block。

2. DiT decoder的输入插入的不是learnable mask token，而是直接插入invisible patch，diffusion loss在所有patch上计算，而不是像MaskDiT那样只根据visible token去预测invisible patch。

3. 这样去掉了MAE，没有了理解就无法辅助生成，所以引入self-distilling模块。encoder的每个token处的最后一层输出再经过一个MLP+softmax预测一个$K$$K$维的分布，以teacher encoder处的预测结果为label计算cross-entropy loss，只计算所有unmask token和class token处的cross-entropy loss。

Patch

PatchDiffusion

Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models

1. 对$x$$x$进行random crop得到$x_{i,j,s}$$x_{i,j,s}$，其中$i,j$$i,j$为左上角坐标，$s$$s$为patch size，对$x_{i,j,s}$$x_{i,j,s}$​进行加噪训练EDM，$i,j,s$$i,j,s$也作为输入条件。

2. EDM only sees local patches and may have not captured the global cross-region dependency between local patches, in other words, the learned scores from nearby patches should form a coherent score map to induce coherent image sampling. To resolve this issue, we propose two strategies: 1) random patch sizes and 2) involving a small ratio of full-size images.

3. 采样时分patch采样后拼在一起。

4. Through Patch Diffusion, we could achieve $\ge 2$$\ge 2$ faster training, while maintaining comparable or better generation quality.

Patch-DM

Patched Denoising Diffusion Models For High-Resolution Image Synthesis

1. Rather than using entire complete images for training, our model only takes patches for training and inference and uses feature collage to systematically combine partial features of neighboring patches.

2. 训练和推导时，第一种方法是将$x_t$$x_{t}$分成patch，每个patch输入模型单独预测；第二种方法是将$x_t$$x_{t}$分成patch，相邻patch输入模型预测公共部分；第三种方法将第二种方法细化到UNet的feature上。

novel diffusion formula

一些具体任务本身就是某种过程，可以设计不同的马尔科夫转移链进行训练。

ShiftDDPMs

ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectories

ContextDiff

Cross-Modal Contextualized Diffusion Models for Text-Guided Visual Generation and Editing

CARD

CARD: Classification and Regression Diffusion Models

公式类似PriorGrad，diffusion model输出regression的值或者classification的概率。

ExposureDiffusion

ExposureDiffusion: Learning to Expose for Low-light Image Enhancement

把照相机图像曝光过程当做一种扩散过程进行建模。

RDDM

Residual Denoising Diffusion Models

类似ResShift。

Beta-Diffusion

Beta Diffusion

Beta分布，优化KL-divergence upper bounds。

others

FDM

Fast Diffusion Model

与SGD建立联系，引入momentum，加快训练和采样。

DDDM

Directly Denoising Diffusion Model

1. DDDMs train the diffusion model conditioned on an es timated target that was generated from previous training iterations of its own.

2. Define $f_{\theta }(x_0,x_t,t)=x_t+{\int }_t^0-\frac{1}{2}\beta (s)[x_s-{\mathrm{\nabla }}_{x_s}\log q(x_s)]ds$$f_{\theta}(x_{0}, x_{t}, t) = x_{t} + \int_{t}^{0} - \frac{1}{2} \beta(s) [x_{s} - \nabla_{x_{s}} \log q(x_{s}) ] ds$ as the solution of the VP PF ODE from initial time $t$$t$ to final time $0$$0$，使用神经网络表示$F_{\theta }(x_0,x_t,t)={\int }_t^0\frac{1}{2}\beta (s)[x_s-{\mathrm{\nabla }}_{x_s}\log q(x_s)]ds$$F_{\theta}(x_{0}, x_{t}, t) = \int_{t}^{0} \frac{1}{2} \beta(s) [x_{s} - \nabla_{x_{s}} \log q(x_{s}) ] ds$，所以$f_{\theta }(x_0,x_t,t)=x_t-F_{\theta }(x_0,x_t,t)$$f_{\theta}(x_{0}, x_{t}, t) = x_{t} - F_{\theta}(x_{0}, x_{t}, t)$。虽然使用了PF ODE，但不需要预训练的score model。

DeeDiff

DeeDiff: Dynamic Uncertainty-Aware Early Exiting for Accelerating Diffusion Model Generation

early exiting策略：The basic assumption of early exiting is that the input samples of the test set are divided into easy and hard samples. The computation for easy samples terminates once some conditions are satisfied.

使用U-ViT，在训练时为每层的输出额外训练一个uncertainty estimation module，用于评估当前层作为输出的不确定度。UEM是一个预测标量的MLP，目标是当前层的输出与最后一层的输出的MSE Loss。

推理时，对于采样的每一步，一旦某层的输出的不确定度低于给定阈值，就将该层输出作为最终输出，达到加速的效果。

DMP

Diffusion Model Patching via Mixture-of-Prompts

1. 为预训练好的DiT的每层block额外训练一组参数$p^i$$p^{i}$，其和输入$x^i$$x^{i}$维度相同，类似positional embedding一样加在$x^i$$x^{i}$上。

2. The same prompts are used for each block throughout the training, thus they will learn knowledge that is agnostic to denoising stages. To patch the model with stage-specific knowledge, we introduce dynamic gating. This mechanism blends prompts in varying proportions based on the noise level of an input image. 学习一个gating网络，$x^i={\text{block}}^i(\sigma (G([t;i]))\odot p^{i-1}+x^{i-1})$$x^{i} = \text{block}^{i}(\sigma(G([t;i])) \odot p^{i-1} + x^{i-1})$。

Compensation

Compensation Sampling for Improved Convergence in Diffusion Models

额外训练一个UNet预测补全项。

CEP

Slight Corruption in Pre-training Data Makes Better Diffusion Models

1. 类似CADS对condition进行操作，但CADS仅在采样时进行操作，二CEP在训练时进行操作。

2. 初步实验：To introduce synthetic corruption into the conditions, we randomly flip the class label into a random class for IN-1K, and randomly swap the text of two sampled image-text pairs for CC3M. As a result, class and text-conditional models pre-trained with slight corruption achieve significantly lower FID and higher IS and CLIP score. More corruption in pre-training can potentially lead to quality and diversity degradation. As degration level increases, almost all metrics first become better and then degrade. However, the degraded measure with more corruption sometimes is still better than the clean ones.

3. 更一般的，we propose to directly add perturbation to the conditional embeddings of DMs, which is termed as conditional embedding perturbation。做法是对condition embedding加一个符合$\mathcal{N}(0,\frac{\gamma }{\sqrt{d}}I)$$\mathcal{N}(0,\frac{\gamma}{\sqrt{d}}I)$的高斯噪声。

SADM

Structure-Guided Adversarial Training of Diffusion Models

除了diffusion loss，用batch内$x_0$$x_{0}$计算两两之间的manifold距离（用预训练编码网络将图像编码为一个向量并计算它们的欧式距离），同时使用预测的${\hat{x}}_0$$\hat{x}_{0}$也计算出batch内两两之间的manifold距离，优化使这两个距离极小化，目的是让diffusion预测出的图像保持和原数据集同样的manifold structure。

如果使用预训练好的编码网络会导致shortcut，所以引入对抗训练，训练编码网络极大化上述两个距离之间的差异（相当于区分fake和real的manifold structure）。

ConPreDiff

Improving Diffusion-Based Image Synthesis with Context Prediction

1. 除了传统的diffusion loss（self-denoising）根据$x_t$$x_{t}$预测所有的$x_{t-1}^i$$x^{i}_{t-1}$，额外使用一个网络根据预测出的$x_{t-1}^i$$x^{i}_{t-1}$预测$x_{t-1}^i$$x^{i}_{t-1}$周围的neighhoods $x_{t-1}^j$$x^{j}_{t-1}$​，the ConPreDiff loss is an upper bound of the negative log likelihood. 这个loss的梯度也会传到self-denoising网络（UNet or Transformer）。

2. 采样时只用self-denoising网络，和传统diffusion model一致。

Time-Dependent

The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks

1. 探索更好的引入timestep embedding的方式。

enhancement of sampling

optimal sampling schedule

DP

Learning to Efficiently Sample from Diffusion Probabilistic Models

RL

Learning to Schedule in Diffusion Probabilistic Models

AYS

Align Your Steps: Optimizing Sampling Schedules in Diffusion Models

LD3

Learning to Discretize Denoising Diffusion ODEs

1. $\xi$$\xi$是一个可学习的、固定长度的、单调递减的、从$T$$T$至$0$$0$的sampling schedule，$\mathrm{\Psi }$$\Psi$是ODE-Solver，${\mathrm{\Psi }}_{\star }$$\Psi_{\star}$是teacher solution，${\mathrm{\Psi }}_{\xi }$$\Psi_{\xi}$是按$\xi$$\xi$ sampling schedule采样的student solution，$\underset{\xi }{min}{\mathcal{L}}_{\text{hard}}=\underset{\xi }{min}{E}_{x_T\sim \mathcal{N}(0,{\sigma }_T^{2}I)}[\text{LPIPS}({\mathrm{\Psi }}_{\xi }(x_T),{\mathrm{\Psi }}_{\star }(x_T))]$$\min_{\xi} \mathcal{L}_{\text{hard}} = \min_{\xi} E_{x_{T} \sim \mathcal{N}(0, \sigma_{T}^{2}I)} [\text{LPIPS}(\Psi_{\xi}(x_{T}), \Psi_{\star}(x_{T})) ]$。

2. Directly optimizing ${\mathcal{L}}_{\text{hard}}$$\mathcal{L}_{\text{hard}}$ could lead to severe underfitting: to minimize the objective, we need to ensure ${\mathrm{\Psi }}_{\star }(x_T)$$\Psi_{\star}(x_{T})$ and ${\mathrm{\Psi }}_{\xi }(x_T)$$\Psi_{\xi}(x_{T})$ for any $x_T$$x_{T}$ , which is hard as we are only allowed to optimize $\xi$$\xi$, which typically contains no more than 20 parameters for student ODE solvers with low NFE. We only require the existence of an input $x_T^{\prime }$$x^{\prime}_{T}$ that is close to $x_T$$x_{T}$ , such that the student’s output given ${\mathrm{\Psi }}_{\xi }(x_T^{\prime })$$\Psi_{\xi}(x^{\prime}_{T})$ matches ${\mathrm{\Psi }}_{\star }(x_T)$$\Psi_{\star}(x_{T})$. Formally, we define $B(x,r{\sigma }_T)=\{x^{\prime }|\parallel x^{\prime }-x{\parallel }_2\le r{\sigma }_T\}$$B(x,r \sigma_{T}) = \{x^{\prime} | \| x^{\prime} -x \|_{2} \le r\sigma_{T} \}$ as the L2 ball of radius $r{\sigma }_T$$r\sigma_{T}$ around $x$$x$，$\underset{\xi }{min}{\mathcal{L}}_{\text{soft}}=\underset{\xi }{min}{E}_{x_T\sim \mathcal{N}(0,{\sigma }_T^{2}I),x_T^{\prime }\in B(x_T,r{\sigma }_T)}[\text{LPIPS}({\mathrm{\Psi }}_{\xi }(x_T^{\prime }),{\mathrm{\Psi }}_{\star }(x_T))]$$\min_{\xi} \mathcal{L}_{\text{soft}} = \min_{\xi} E_{x_{T} \sim \mathcal{N}(0, \sigma_{T}^{2}I), x^{\prime}_{T} \in B(x_{T},r \sigma_{T})} [\text{LPIPS}(\Psi_{\xi}(x^{\prime}_{T}), \Psi_{\star}(x_{T})) ]$。

Distillation-based

大致分为四类：Direct Distillation, Progressive Distillation，Adaversarial Distillation，Score Distillation（DI），Consistency Distillation

DenoisingStudent

Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed

1. direct distillation，$\mathcal{L}=\frac{1}{2}{E}_{x_T}[\parallel {\mathcal{F}}_{\text{student}}(x_T)-{\mathcal{F}}_{\text{teacher}}(x_T){\parallel }_2^2]$$\mathcal{L} = \frac{1}{2} E_{x_{T}}[\| \mathcal{F}_{\text{student}}(x_{T}) - \mathcal{F}_{\text{teacher}}(x_{T}) \|^{2}_{2}]$，其中${\mathcal{F}}_{\text{student}}(x_T)$$\mathcal{F}_{\text{student}}(x_{T})$是student一步生成的样本，${\mathcal{F}}_{\text{teacher}}(x_T)$$\mathcal{F}_{\text{teacher}}(x_{T})$是teacher多步DDIM生成的样本。

2. 本质是使用teacher构造$(x_T,x_0)$$(x_{T}, x_{0})$ pair数据集训练一步生成的student。

Diffusion2GAN

1. SDXL

2. We can significantly improve the quality of direct distillation by (1) scaling up the size of the ODE pair dataset and (2) using a perceptual loss, not MSE loss.

3. 在SDXL的latent上重新训练一个VGG网络，优化student生成的latent和teacher生成的latent之间的LPIPS loss。

4. 除了LPIPS loss，还是用对抗训练，使用类似GigaGAN的multi-scale discriminator。

PD

Progressive Distillation for Fast Sampling of Diffusion Models

1. 训练student一步采样模拟teacher多步采样的效果。

2. v-prediction：在distillation时$ϵ$$\epsilon$-prediction参数法就不再适用了，因为DDIM采样的每一步都要先计算${\hat{x}}_{\theta }=\frac{1}{\sqrt{{\overline{\alpha }}_t}}(x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t))$$\hat{x}_{\theta} = \frac{1}{\sqrt{\bar{\alpha}_{t}}}(x_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}(x_{t},t))$，在最初的几步时$\sqrt{{\overline{\alpha }}_t}\to 0$$\sqrt{\bar{\alpha}_{t}} \rightarrow 0$，所以${ϵ}_{\theta }$$\epsilon_{\theta}$的微小变化都会被放大，极端情况下，当distillation为1步时，该公式就没有意义了，所以需要重新找到一种参数方法，使${\hat{x}}_{\theta }$$\hat{x}_{\theta}$在任何$SNR$$SNR$时都比较稳定。考虑到$(\sqrt{{\overline{\alpha }}_t}{)}^2+(\sqrt{1-{\overline{\alpha }}_t}{)}^2=1$$(\sqrt{\bar{\alpha}_{t}})^{2} + (\sqrt{1-\bar{\alpha}_{t}})^{2}=1$，所以可以设$\sqrt{{\overline{\alpha }}_t}=cos(\varphi )$$\sqrt{\bar{\alpha}_{t}}=cos(\phi)$，$\sqrt{1-{\overline{\alpha }}_t}=sin(\varphi )$$\sqrt{1 - \bar{\alpha}_{t}} = sin(\phi)$，$x_{\varphi }=cos(\varphi )x_0+sin(\varphi )ϵ$$x_{\phi} = cos(\phi) x_{0} + sin(\phi)\epsilon$，定义$v_{\varphi }=\frac{d{x}_{\varphi }}{d\varphi }=cos(\varphi )ϵ-sin(\varphi )x_0$$v_{\phi} = \frac{d x_{\phi}}{d \phi} = cos(\phi)\epsilon - sin(\phi)x_{0}$，所以有$x_0=\frac{cos(\varphi )ϵ-v_{\varphi }}{sin(\varphi )}=\frac{cos(\varphi )\frac{x_{\varphi }-cos(\varphi )x_0}{sin(\varphi )}-v_{\varphi }}{sin(\varphi )}=\frac{cos(\varphi )x_{\varphi }}{si{n}^2(\varphi )}-\frac{co{s}^2(\varphi )}{si{n}^2(\varphi )}{x}_0-\frac{v_{\varphi }}{sin(\varphi )}$$x_{0} = \frac{cos(\phi)\epsilon - v_{\phi}}{sin(\phi)} = \frac{cos(\phi)\frac{x_{\phi}-cos(\phi)x_{0}}{sin(\phi)} - v_{\phi}}{sin(\phi)} = \frac{cos(\phi)x_{\phi}}{sin^{2}(\phi)} -\frac{cos^{2}(\phi)}{sin^{2}(\phi)}x_{0} -\frac{v_{\phi}}{sin(\phi)}$，$(si{n}^2(\varphi )+co{s}^2(\varphi ))x_0=x_0=cos(\varphi )x_{\varphi }-sin(\varphi )v_{\varphi }$$(sin^{2}(\phi)+cos^{2}(\phi))x_{0} = x_{0} = cos(\phi)x_{\phi}-sin(\phi)v_{\phi}$，同理可得$ϵ=sin(\varphi )x_{\varphi }+cos(\varphi )v_{\varphi }$$\epsilon = sin(\phi)x_{\phi} + cos(\phi)v_{\phi}$，如果使用网络$v_{\mathrm{\Omega }}(x_{{\varphi }_t},{\varphi }_t)$$v_{\Omega}(x_{\phi_{t}},\phi_{t})$预测$v_{{\varphi }_t}$$v_{\phi_{t}}$，则等价的两种参数法分别为${\hat{x}}_{\mathrm{\Omega }}=cos({\varphi }_t)x_{{\varphi }_t}-sin({\varphi }_t)v_{\mathrm{\Omega }}(x_{{\varphi }_t},{\varphi }_t)$$\hat{x}_{\Omega} = cos(\phi_{t})x_{\phi_{t}}-sin(\phi_{t})v_{\Omega}(x_{\phi_{t}},\phi_{t})$，${ϵ}_{\mathrm{\Omega }}=sin({\varphi }_t)x_{{\varphi }_t}+cos({\varphi }_t)v_{\mathrm{\Omega }}(x_{{\varphi }_t},{\varphi }_t)$$\epsilon_{\Omega} = sin(\phi_{t})x_{\phi_{t}} + cos(\phi_{t})v_{\Omega}(x_{\phi_{t}},\phi_{t})$，这样${\hat{x}}_{\mathrm{\Omega }}$$\hat{x}_{\Omega}$就很稳定。重写DDIM采样公式可得，$x_{{\varphi }_s}=cos({\varphi }_s){\hat{x}}_{\mathrm{\Omega }}+sin({\varphi }_s){ϵ}_{\mathrm{\Omega }}=cos({\varphi }_s)[cos({\varphi }_t)x_{{\varphi }_t}-sin({\varphi }_t)v_{\mathrm{\Omega }}]+sin({\varphi }_s)[sin({\varphi }_t)x_{{\varphi }_t}+cos({\varphi }_t)v_{\mathrm{\Omega }}]$$x_{\phi_{s}} = cos(\phi_{s}) \hat{x}_{\Omega} + sin(\phi_{s})\epsilon_{\Omega} = cos(\phi_{s}) \left[ cos(\phi_{t})x_{\phi_{t}}-sin(\phi_{t})v_{\Omega} \right] + sin(\phi_{s}) \left[ sin(\phi_{t})x_{\phi_{t}} + cos(\phi_{t})v_{\Omega} \right]$，经过化简可得$x_{{\varphi }_s}=cos({\varphi }_s-{\varphi }_t)x_{{\varphi }_t}+sin({\varphi }_s-{\varphi }_t)v_{\mathrm{\Omega }}$$x_{\phi_{s}} = cos(\phi_{s}-\phi_{t})x_{\phi_{t}} + sin(\phi_{s}-\phi_{t})v_{\Omega}$，即$x_{{\varphi }_t-\mathrm{\Delta }}=cos(-\mathrm{\Delta })x_{{\varphi }_t}+sin(-\mathrm{\Delta })v_{\theta }=cos(\mathrm{\Delta })x_{{\varphi }_t}-sin(\mathrm{\Delta })v_{\mathrm{\Omega }}$$x_{\phi_{t} - \Delta} = cos(- \Delta)x_{\phi_{t}} + sin(- \Delta)v_{\theta} = cos(\Delta)x_{\phi_{t}} - sin(\Delta)v_{\Omega}$，如图所示的三角关系，DDIM采样相当于从$ϵ$$\epsilon$开始向切线方向不断移动。使用该参数法重新训练diffusion model，效果也很好。distillation即为使用student网络一步预测出的$v_{\mathrm{\Omega }}$$v_{\Omega}$计算出的${\hat{x}}_{\mathrm{\Omega }}$$\hat{x}_{\Omega}$拟合teacher网络多步预测出的${\hat{x}}_{\theta }$$\hat{x}_{\theta}$（不管如何参数化，最终都转换为预测出的${\hat{x}}_0$$\hat{x}_{0}$之间的拟合）。